

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>The ClientRequest pipeline &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/underscore.js"></script>
        <script src="../../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="PoseidonStore" href="../poseidonstore/" />
    <link rel="prev" title="osd" href="../osd/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../../internals/">Ceph 内幕</a> &raquo;</li>
        
          <li><a href="../">Crimson developer documentation</a> &raquo;</li>
        
      <li>The <code class="docutils literal notranslate"><span class="pre">ClientRequest</span></code> pipeline</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../../_sources/dev/crimson/pipeline.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../">
          

          
            
            <img src="../../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../developer_guide/">开发者指南</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../internals/">Ceph 内幕</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../blkin/">Tracing Ceph With LTTng</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../blkin/#tracing-ceph-with-blkin">Tracing Ceph With Blkin</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../bluestore/">BlueStore Internals</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cache-pool/">Cache pool</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../ceph_krb_auth/">如何配置好 Ceph Kerberos 认证的详细文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-mirroring/">CephFS Mirroring</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-reclaim/">CephFS Reclaim Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephfs-snapshots/">CephFS 快照</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephx/">Cephx</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cephx_protocol/">Cephx 认证协议详细阐述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../config/">配置管理系统</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../config-key/">config-key layout</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../context/">CephContext</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../continuous-integration/">Continuous Integration Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../corpus/">资料库结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cpu-profiler/">Oprofile 的安装</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../cxx/">C++17 and libstdc++ ABI</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../deduplication/">去重</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../delayed-delete/">CephFS delayed deletion</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dev_cluster_deployement/">开发集群的部署</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dev_cluster_deployement/#id5">在同一机器上部署多套开发集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../development-workflow/">开发流程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../documenting/">为 Ceph 写作文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../encoding/">序列化（编码、解码）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../erasure-coded-pool/">纠删码存储池</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../file-striping/">File striping</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../freebsd/">FreeBSD Implementation details</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../generatedocs/">Ceph 文档的构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../health-reports/">Health Reports</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../iana/">IANA 号</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../kubernetes/">Hacking on Ceph in Kubernetes with Rook</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../libs/">库体系结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../logging/">集群日志的用法</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../logs/">调试日志</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../macos/">在 MacOS 上构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../messenger/">Messenger notes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-bootstrap/">Monitor bootstrap</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-elections/">Monitor Elections</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-on-disk-formats/">ON-DISK FORMAT</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mon-osdmap-prune/">FULL OSDMAP VERSION PRUNING</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../msgr2/">msgr2 协议（ msgr2.0 和 msgr2.1 ）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../network-encoding/">Network Encoding</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../network-protocol/">网络协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../object-store/">对象存储架构概述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../osd-class-path/">OSD class path issues</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../peering/">互联</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf/">Using perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf_counters/">性能计数器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../perf_histograms/">Perf histograms</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../placement-group/">PG （归置组）说明</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../quick_guide/">开发者指南（快速）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rados-client-protocol/">RADOS 客户端协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-diff/">RBD 增量备份</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-export/">RBD Export &amp; Import</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../rbd-layering/">RBD Layering</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../release-checklists/">Release checklists</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../release-process/">Ceph Release Process</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../seastore/">SeaStore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../sepia/">Sepia 社区测试实验室</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../session_authentication/">Session Authentication for the Cephx Protocol</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../testing/">测试笔记</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../versions/">Public OSD Version</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../vstart-ganesha/">NFS CephFS-RGW Developer Guide</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../wireshark/">Wireshark Dissector</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../zoned-storage/">Zoned Storage Support</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../osd_internals/">OSD 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../mds_internals/">MDS 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../radosgw/">RADOS 网关开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../ceph-volume/">ceph-volume 开发者文档</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">Crimson developer documentation</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../crimson/">crimson</a></li>
<li class="toctree-l3"><a class="reference internal" href="../error-handling/">error handling</a></li>
<li class="toctree-l3"><a class="reference internal" href="../osd/">osd</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">The <code class="docutils literal notranslate"><span class="pre">ClientRequest</span></code> pipeline</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#comparison-with-the-classical-osd">Comparison with the classical OSD</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../poseidonstore/">PoseidonStore</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="the-clientrequest-pipeline">
<h1>The <code class="docutils literal notranslate"><span class="pre">ClientRequest</span></code> pipeline<a class="headerlink" href="#the-clientrequest-pipeline" title="Permalink to this headline">¶</a></h1>
<p>In crimson, exactly like in the classical OSD, a client request has data and
ordering dependencies which must be satisfied before processing (actually
a particular phase of) can begin. As one of the goals behind crimson is to
preserve the compatibility with the existing OSD incarnation, the same semantic
must be assured. An obvious example of such data dependency is the fact that
an OSD needs to have a version of OSDMap that matches the one used by the client
(<code class="docutils literal notranslate"><span class="pre">Message::get_min_epoch()</span></code>).</p>
<p>If a dependency is not satisfied, the processing stops. It is crucial to note
the same must happen to all other requests that are sequenced-after (due to
their ordering requirements).</p>
<p>There are a few cases when the blocking of a client request can happen.</p>
<blockquote>
<div><dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">ClientRequest::ConnectionPipeline::await_map</span></code></dt><dd><p>wait for particular OSDMap version is available at the OSD level</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">ClientRequest::ConnectionPipeline::get_pg</span></code></dt><dd><p>wait a particular PG becomes available on OSD</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">ClientRequest::PGPipeline::await_map</span></code></dt><dd><p>wait on a PG being advanced to particular epoch</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">ClientRequest::PGPipeline::wait_for_active</span></code></dt><dd><p>wait for a PG to become <em>active</em> (i.e. have <code class="docutils literal notranslate"><span class="pre">is_active()</span></code> asserted)</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">ClientRequest::PGPipeline::recover_missing</span></code></dt><dd><p>wait on an object to be recovered (i.e. leaving the <code class="docutils literal notranslate"><span class="pre">missing</span></code> set)</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">ClientRequest::PGPipeline::get_obc</span></code></dt><dd><p>wait on an object to be available for locking. The <code class="docutils literal notranslate"><span class="pre">obc</span></code> will be locked
before this operation is allowed to continue</p>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">ClientRequest::PGPipeline::process</span></code></dt><dd><p>wait if any other <code class="docutils literal notranslate"><span class="pre">MOSDOp</span></code> message is handled against this PG</p>
</dd>
</dl>
</div></blockquote>
<p>At any moment, a <code class="docutils literal notranslate"><span class="pre">ClientRequest</span></code> being served should be in one and only one
of the phases described above. Similarly, an object denoting particular phase
can host not more than a single <code class="docutils literal notranslate"><span class="pre">ClientRequest</span></code> the same time. At low-level
this is achieved with a combination of a barrier and an exclusive lock.
They implement the semantic of a semaphore with a single slot for these exclusive
phases.</p>
<p>As the execution advances, request enters next phase and leaves the current one
freeing it for another <code class="docutils literal notranslate"><span class="pre">ClientRequest</span></code> instance. All these phases form a pipeline
which assures the order is preserved.</p>
<p>These pipeline phases are divided into two ordering domains: <code class="docutils literal notranslate"><span class="pre">ConnectionPipeline</span></code>
and <code class="docutils literal notranslate"><span class="pre">PGPipeline</span></code>. The former ensures order across a client connection while
the latter does that across a PG. That is, requests originating from the same
connection are executed in the same order as they were sent by the client.
The same applies to the PG domain: when requests from multiple connections reach
a PG, they are executed in the same order as they entered a first blocking phase
of the <code class="docutils literal notranslate"><span class="pre">PGPipeline</span></code>.</p>
<div class="section" id="comparison-with-the-classical-osd">
<h2>Comparison with the classical OSD<a class="headerlink" href="#comparison-with-the-classical-osd" title="Permalink to this headline">¶</a></h2>
<p>As the audience of this document are Ceph Developers, it seems reasonable to
match the phases of crimson’s <code class="docutils literal notranslate"><span class="pre">ClientRequest</span></code> pipeline with the blocking
stages in the classical OSD. The names in the right column are names of
containers (lists and maps) used to implement these stages. They are also
already documented in the <code class="docutils literal notranslate"><span class="pre">PG.h</span></code> header.</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 51%" />
<col style="width: 49%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>crimson</p></th>
<th class="head"><p>ceph-osd waiting list</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">ConnectionPipeline::await_map</span></code>
<code class="docutils literal notranslate"><span class="pre">ConnectionPipeline::get_pg</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">OSDShardPGSlot::waiting</span></code> and
<code class="docutils literal notranslate"><span class="pre">OSDShardPGSlot::waiting_peering</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">PGPipeline::await_map</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_map</span></code></p></td>
</tr>
<tr class="row-even"><td rowspan="3"><p><code class="docutils literal notranslate"><span class="pre">PGPipeline::wait_for_active</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_peered</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_flush</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_active</span></code></p></td>
</tr>
<tr class="row-odd"><td><p>To be done (<code class="docutils literal notranslate"><span class="pre">PG_STATE_LAGGY</span></code>)</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_readable</span></code></p></td>
</tr>
<tr class="row-even"><td><p>To be done</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_scrub</span></code></p></td>
</tr>
<tr class="row-odd"><td rowspan="2"><p><code class="docutils literal notranslate"><span class="pre">PGPipeline::recover_missing</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_unreadable_object</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_degraded_object</span></code></p></td>
</tr>
<tr class="row-odd"><td><p>To be done (proxying)</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">PG::waiting_for_blocked_object</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">PGPipeline::get_obc</span></code></p></td>
<td><p><em>obc rwlocks</em></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">PGPipeline::process</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">PG::lock</span></code> (roughly)</p></td>
</tr>
</tbody>
</table>
<p>As the last word it might be worth to emphasize that the ordering implementations
in both classical OSD and in crimson are stricter than a theoretical minimum one
required by the RADOS protocol. For instance, we could parallelize read operations
targeting the same object at the price of extra complexity but we don’t – the
simplicity has won.</p>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../poseidonstore/" class="btn btn-neutral float-right" title="PoseidonStore" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../osd/" class="btn btn-neutral float-left" title="osd" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>